Describing UI Screenshots in Natural Language

Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by global precedence effect create informative descriptions UIs, starting an overview and then providing fine-grained about most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, generation (NLG). provides up three different granularity levels that, together, what is do it. found that are highly readable, perceived accurately score similarly human-generated UI descriptions. available as open-source software.